Wasm Edge #1864

Harrm · 2023-11-10T11:29:22Z

Referenced issues

Description of the Change

Benefits

Possible Drawbacks

Usage Examples or Tests

Alternate Designs

…t-pruner-start

…arate-nodes-and-values

igor-egorov

💪

igor-egorov · 2023-11-24T08:52:34Z

core/runtime/binaryen/runtime_external_interface.cpp

@@ -101,6 +96,7 @@ namespace {
  template <auto mf>
  wasm::Literal callHostApiFunc(kagome::host_api::HostApi *host_api,
                                const wasm::LiteralList &arguments) {
+    std::cout << "Call Host method " << __PRETTY_FUNCTION__  << "\n";


Still actual

core/runtime/wasm_edge/wrappers.hpp

core/runtime/wasm_edge/module_factory_impl.cpp

core/runtime/wasm_edge/register_host_api.hpp

core/application/impl/app_configuration_impl.cpp

turuslan · 2023-11-24T10:21:12Z

core/host_api/impl/child_storage_extension.cpp

@@ -9,10 +9,11 @@
 #include <tuple>
 #include <utility>

+#include <fmt/std.h>


"fmt/std.h" formatter<std::error_code> = "{category}:{enum}" (no message) conflicts with "qtils/error.hpp" formatter<std::error_code> = "{category}:{enum}:{message}".
Should we replace "qtils/error.hpp" formatter with #include <fmt/std.h>?

Also conflicts with <libp2p/outcome/outcome.hpp>

qtils/error.hpp looks more descriptive.

turuslan · 2023-11-24T10:32:44Z

core/storage/trie/trie_storage_backend.hpp

   * along with root hash storing logic from the trie db component
   */
-  class TrieStorageBackend : public BufferStorage {
+  class TrieNodeStorageBackend : public BufferStorage {


(maybe changes related to #1770 from wasmedge pr)
Use BufferStorage directly, or use wrapper instead of inheritance.
#1770 (comment)

Suggested change

class TrieNodeStorageBackend : public BufferStorage {

using TrieNodeStorageBackend = BufferStorage;

struct TrieValueStorageBackend = BufferStorage;

Suggested change

class TrieNodeStorageBackend : public BufferStorage {

struct TrieNodeStorageBackend { std::shared_ptr<BufferStorage> v; };

struct TrieValueStorageBackend { std::shared_ptr<BufferStorage> v; };

BufferStorage is good simple interface.
To reuse existing BufferStorage implementations for new interface you need to create proxy wrapper class inheriting new interface (new interface calls corresponding BufferStorage) or just reinterpret_cast.
TrieNodeStorageBackend/TrieValueStorageBackend don't add any functions to BufferStorage, so they must not inherit from it and just wrap pointer.

turuslan · 2023-11-24T10:38:45Z

core/storage/trie/impl/trie_storage_backend_impl.hpp

+  class TrieStorageBackendImpl : public TrieNodeStorageBackend,
+                                 public TrieValueStorageBackend {


Does this strange diamond inheritance require virtual inheritance?
(Remove this class, use BufferStorage directly)

These are just interfaces.

These are ambigous/conflicting interfaces.
Without virtual inheritance this class will contain BufferStorage instance twice (for each non-virtual inheriting subclass).

BufferStorage is abstract base class and wont be instantiated.

BufferStorage is abstract base class and wont be instantiated.

You can't create BufferStorage variable because it's vtable is empty,
but subclass always contains/stores superclass (struct SubClass { SuperClass superclass; }).

turuslan · 2023-11-24T10:42:36Z

core/storage/spaces.hpp

@@ -20,6 +20,7 @@ namespace kagome::storage {
    kBlockBody,
    kJustification,
    kTrieNode,
+    kTrieValue,


May use same kTrieNode space with prefix (both!)
node_db = MapPrefix{"n", db.getSpace(kTrieNode)}
value_db = MapPrefix{"v", db.getSpace(kTrieNode)}

core/runtime/wasm_edge/module_factory_impl.cpp

xDimon · 2023-11-26T16:34:04Z

cmake/Hunter/hunter-gate-url.cmake

@@ -1,5 +1,5 @@
 HunterGate(
-  URL  https://github.com/qdrvm/hunter/archive/refs/tags/v0.23.257-qdrvm11.tar.gz
-  SHA1 b2a69ae501bdc99006fb86e55930640004468556
+  URL "https://github.com/qdrvm/hunter/archive/heads/feature/wasm-edge.zip"


Hunter with wasm-edge should be released before merge

core/application/app_configuration.hpp

core/injector/CMakeLists.txt

xDimon · 2023-11-26T18:33:38Z

core/injector/application_injector.cpp

@@ -156,6 +157,16 @@
 #include "runtime/runtime_api/impl/session_keys_api.hpp"
 #include "runtime/runtime_api/impl/tagged_transaction_queue.hpp"
 #include "runtime/runtime_api/impl/transaction_payment_api.hpp"
+
+#if KAGOME_WASM_COMPILER_WASM_EDGE == 1


Or #ifdef

core/runtime/binaryen/memory_impl.cpp

core/runtime/wasm_edge/module_factory_impl.cpp

core/runtime/wasm_edge/module_factory_impl.hpp

xDimon · 2023-11-26T19:45:18Z

core/runtime/wasm_edge/register_host_api.hpp

+  void register_host_api(WasmEdge_ModuleInstanceContext *instance) {
+    BOOST_ASSERT(instance);
+    // clang-format off
+    REGISTER_HOST_METHOD(void, ext_allocator_free_version_1, WasmPointer)


Could you extract list of ext_ function to file common for WAVM and WasmEdge?

Do you want it to look like this?

/// host_api_macro.hpp // no "#pragma once" HOST_API_MACRO(void, ext_allocator_free_version_1, WasmPointer) HOST_API_MACRO(int32_t, ext_allocator_malloc_version_1, int32_t) ... /// wavm.hpp #define HOST_API_MACRO(...) wavm.add(...) #include "host_api_macro.hpp" /// wasmedge.hpp #define HOST_API_MACRO(...) wasmedge.add(...) #include "host_api_macro.hpp"

or try to make tuple-like template?

template <auto...> struct TemplateTuple {}; using HostApiTuple = TemplateTuple< &HostApi::ext_allocator_free_version_1, &HostApi::ext_allocator_malloc_version_1, ... >;

Yes, I'll look into it. Maybe in this PR, maybe make an issue.

core/runtime/wasm_edge/wrappers.hpp

xDimon · 2023-11-26T19:55:58Z

core/storage/trie/impl/trie_storage_backend_impl.hpp

+  class TrieStorageBackendImpl : public TrieNodeStorageBackend,
+                                 public TrieValueStorageBackend {


BufferStorage is abstract base class and wont be instantiated.

core/runtime/wasm_edge/module_factory_impl.cpp

turuslan · 2023-11-28T09:56:41Z

core/runtime/wasm_edge/memory_impl.hpp

+    common::BufferView loadN(WasmPointer addr, WasmSize n) const override {
+      BOOST_ASSERT(n > 0);
+      auto ptr = WasmEdge_MemoryInstanceGetPointer(mem_instance_, addr, n);
+      BOOST_ASSERT(ptr);


throw (or you will get NullPointerException)

Memory methods may be called by the host outside of the runtime call, where we catch exceptions, so given that most our code is exception-free I'm not sure it's a good idea to throw here. I think we're gonna need to tackle this problem separately and produce a uniform solution for all WASM executors in kagome.

loadN (and all callers) can return outcome.
host api can return outcome or throw with .value(), and host api wrapper function will check outcome or catch exceptions

core/runtime/wasm_edge/register_host_api.hpp

…ure/wasm-edge

Alexander Krutikov and others added 30 commits December 24, 2021 22:18

wasm-edge dep

7dce18d

wasmedge host api

96cb891

wasmedge compile host api

72580e0

wasmedge memory

0a8981b

fix wasmedge memory

d931648

add wasmedge module instance

8245195

add wasmedge instance environment factory

1f6d8e4

add wasmedge option

0c117ec

add wasmedge module

ec6be5e

update wasmedge, remove WAVM temporarily

7a84fe8

implement compiling wasmedge

d72b0a7

fix

3ca7256

fix2

3a344c0

bump WasmEdge to 0.9.0

a56ba2d

Remove weird optional locks

24f482d

Fast pruner startup

b05e9c7

Add option for thorough pruning

006a4ce

Merge branch 'master' of github.com:soramitsu/kagome into feature/fas…

391ebe5

…t-pruner-start

Separate trie nodes and values into separate spaces

8f33d1f

Merge branch 'master' of github.com:soramitsu/kagome into feature/sep…

404e269

…arate-nodes-and-values

Fix tests

9c9ba57

WasmEdge integration in progress

dc44749

Temporarily exclude WAVM

04e4964

Working version of WasmEdge

96a7e18

Merge with old wasm-edge accidentally

afe9325

Merge with master

fcbc8f6

Merge with master

65fc416

Separate WasmEdge and WAVM at compile time

5e972bf

Merge with master

1aecebb

Various fixes for WasmEdge

7d1a4cd

In progress fixing tests

c72865b

igor-egorov reviewed Nov 24, 2023

View reviewed changes

turuslan reviewed Nov 24, 2023

View reviewed changes

xDimon reviewed Nov 26, 2023

View reviewed changes

Harrm added 4 commits November 27, 2023 12:13

Resolving review comments

d81e901

Review fixes

dc7e853

Merge

45fe2b3

Review fixes

2e326ad

turuslan reviewed Nov 28, 2023

View reviewed changes

Remove trie node and value separation

3b8d2bb

Harrm marked this pull request as ready for review November 28, 2023 13:18

Harrm added 12 commits November 28, 2023 16:19

Merge branch 'master' into feature/wasm-edge

0e4061b

Hunterized LLVM

2616498

Merge branch 'feature/wasm-edge' of github.com:qdrvm/kagome into feat…

53a2425

…ure/wasm-edge

Fixing hunter

71a326a

Fix wasmedge hunterization

04413c1

Remove current_host_api mentions

daf682e

Remove current_host_api mentions

83e1824

Switch macos ci to wasmedge

442a85e

Fix WAVM build

b446695

Fix tests

2575305

Fix trie backend test

97b75d6

Merge branch 'master' into feature/wasm-edge

1371900

igor-egorov approved these changes Dec 1, 2023

View reviewed changes

turuslan approved these changes Dec 1, 2023

View reviewed changes

Harrm added 2 commits December 1, 2023 15:06

Throw host_api exceptions

4094bd2

Merge branch 'feature/wasm-edge' of github.com:qdrvm/kagome into feat…

9f35015

…ure/wasm-edge

Harrm enabled auto-merge (squash) December 1, 2023 12:07

Harrm merged commit ee15e3d into master Dec 1, 2023
10 checks passed

Harrm deleted the feature/wasm-edge branch December 1, 2023 12:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wasm Edge #1864

Wasm Edge #1864

Harrm commented Nov 10, 2023

igor-egorov left a comment

igor-egorov Nov 24, 2023

xDimon Nov 26, 2023

turuslan Nov 24, 2023

turuslan Nov 28, 2023

Harrm Nov 29, 2023

turuslan Nov 24, 2023

Harrm Nov 26, 2023

turuslan Nov 26, 2023

turuslan Nov 24, 2023

Harrm Nov 26, 2023

turuslan Nov 26, 2023

xDimon Nov 26, 2023

turuslan Nov 27, 2023

turuslan Nov 24, 2023 •

edited

Loading

xDimon Nov 26, 2023

xDimon Nov 26, 2023 •

edited by turuslan

Loading

xDimon Nov 26, 2023

turuslan Nov 27, 2023 •

edited

Loading

Harrm Nov 28, 2023

xDimon Nov 26, 2023

turuslan Nov 28, 2023

Harrm Dec 1, 2023

turuslan Dec 1, 2023

	class TrieNodeStorageBackend : public BufferStorage {
	using TrieNodeStorageBackend = BufferStorage;
	struct TrieValueStorageBackend = BufferStorage;

	class TrieNodeStorageBackend : public BufferStorage {
	struct TrieNodeStorageBackend { std::shared_ptr<BufferStorage> v; };
	struct TrieValueStorageBackend { std::shared_ptr<BufferStorage> v; };

		class TrieStorageBackendImpl : public TrieNodeStorageBackend,
		public TrieValueStorageBackend {

Wasm Edge #1864

Wasm Edge #1864

Conversation

Harrm commented Nov 10, 2023

Referenced issues

Description of the Change

Benefits

Possible Drawbacks

Usage Examples or Tests

Alternate Designs

igor-egorov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turuslan Nov 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xDimon Nov 26, 2023 • edited by turuslan Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turuslan Nov 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turuslan Nov 24, 2023 •

edited

Loading

xDimon Nov 26, 2023 •

edited by turuslan

Loading

turuslan Nov 27, 2023 •

edited

Loading